Devices and methods using machine learning for surveillance and granting of privileges

ABSTRACT

A method and system where a first subsystem makes observations and performs surveillance using sensors in a mode that conserves a resource such as power, data transmission band width or processing cycles. This is accomplished by reducing illumination, pixel count, sampling rate or other functions that result in a limited granularity or data collection rate. A machine model is applied to the limited data and, when it evaluates to a suitable result or a prediction of an interesting condition, another subsystem or the same subsystem in a different mode collects data at a finer granularity with a higher data collection size or rate and evaluates that data to determine the nature of the first evaluation. The machine model may be trained in stages on a large scale server and on a small field processor. Data from the sensor may be used for training to improve the second step.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation and claims priority to U.S. patentapplication Ser. No. 16/865,396 filed May 3, 2020 now pending which is acontinuation of U.S. patent application Ser. No. 16/390,049 filed Apr.22, 2019 now U.S. Pat. No. 10,657,444 issued May 19, 2020 which claimspriority to U.S. provisional patent application 62/690,367 filed Jun.27, 2018. U.S. patent application Ser. No. 16/390,049 is acontinuation-in-part of U.S. patent application Ser. No. 15/885,684filed Jan. 31, 2018, now U.S. Pat. No. 10,282,668 issued May 7, 2019.U.S. patent application Ser. No. 15/885,684 is a continuation-in-part ofU.S. patent application Ser. No. 15/453,996 filed Mar. 9, 2017, now U.S.Pat. No. 9,921,068.

Each patent application identified above is incorporated here byreference in its entirety to provide continuity of disclosure. Where adefinition or use of a term in a reference, which is incorporated byreference herein, is inconsistent or contrary to the definition of thatterm provided herein, the definition of that term provided hereinapplies and the definition of that term in the reference does not apply.

Furthermore, where a definition or use of a term in a reference, whichis included in material in a section marked as material from the parentapplication, is inconsistent or contrary to the definition or use of theterm in a corresponding section marked as material for the currentapplication, the definition of the term for the current applicationapplies and the definition of that term in the parent application doesnot apply.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

BACKGROUND OF THE INVENTION Field of the Current Invention

The present invention is in the field of surveillance monitoring.

Background Concerning the Need for the Current Invention

Whatever the nature of the sensors used for surveillance purposes, therecan be a need for continuous vigilance. Using the full power of cameras,microphones and other detectors can use substantial resources which maynot be available continuously. Used in a more limited mode thatconserves the resources they may be sufficient to detect situationswhere a temporary increase in resource usage is required. In thislimited recognition of the situation needs to be automated.

BRIEF SUMMARY OF THE INVENTION Brief Summary of the GrandparentInvention

The parent invention envisions devices and method using the devices toescape from a venue when a threat is detected. A processor constructs amodel of a venue in its memory based on first information which isavailable prior to entry of a person to be protected from potentialhazards. The model is trained by machine learning methods and receivesfurther training after the person enters the venue.

The model, thereby, takes into account the location of the person andinformation gathered by sensors about conditions on entry. Otherpre-training can concern methods of escape from typical venues andbehaviors by persons in the presence of hazards.

Observed conditions are evaluated by a processor in accordance with themodel and the device generates an escape plan when a hazard is detected.The escape plan is displayed or transmitted to the protected person tofacilitate escape from the threatened venue. The protected personthereby has an improved chance of successful escape in spite ofdifficulties that may appear from the threat or the conditions ofescape. Warnings of a hazard may be detected and transmitted separatelyfrom the plan of escape.

Brief Summary of the Parent Invention

The current invention claimed in this application concerns devices andmethods to determine whether or not physical elements present in asituation are compliant with a set of rules by using a machine learningsystem to test compliance from data gathered by one or more sensors. Themachine learning system is trained on one processor with data concerningobjects and behaviors indicating compliance with the rules. Theresulting trained model is downloaded to another processor for field usein determining compliance in specific cases. There is further trainingof the model with specific information concerning the context of thecase to be examined and then data from the sensor is used to determinecompliance.

The inputs to the model for the determination come at least from thesensor and perhaps from servers or other sources as well. The output ofthe evaluation of the model is used to transmit the determination or tocontrol granting of a privilege that is intended to be conditional oncompliance with the rules. The additional inputs may be selected on thebasis of a previous or partial evaluation of the model or on the basisof data from the sensor. In some cases, more than two processors may beused. For, example one processor may do the pre-training, one thetraining concerning specific circumstances for the case to be tested,and a third to do the evaluation.

Brief Summary of the Current Invention

The current invention is a system or method to allow observations ingreater detail or granularity when conditions indicate the desirabilityof having this detail, but observing at other times with reducedresource usage. The steps to implement

this are (1) train model with sensors to identify conditions which mayhave a need for observation with fine granularity, (2) observe at acoarse granularity saving resources, (3) process observations with thetrained model which allows determination that there is an increase inprobability of an interesting target, (4) if model applied to inputsfrom observation at coarse granularity producesappropriate output put the observation device in a fine granularity mode(5) observe at fine granularity.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The features and advantages of the various embodiments disclosed hereinwill be better understood with respect to the drawing in which:

Figures from the Parent Application

FIG. 1 through FIG. 10 appear in the grandparent and parentapplications; FIG. 10 through 17 are new to the parent Application.

FIG. 1 is plan diagram of a protected person entering a venue with anescape device.

FIG. 2 is plan diagram of a protected person leaving a venue with theguidance of an escape device after a hazard is detected.

FIG. 3 shows a front view of a typical embodiment of an escape guidingdevice

FIG. 4 shows a back view of the device of FIG. 3.

FIG. 5 shows a venue on board an aircraft.

FIG. 6 shows an escape device as part of the equipment of variouspersons.

FIG. 7 shows an embodiment of an escape device in a smart phone.

FIG. 8 shows an embodiment of an escape device in a vehicle.

FIG. 9 shows the steps and structure of the basic information flow ofthe device and its training.

FIG. 10 shows a diagram for process of developing software for an escapedevice.

FIG. 11 shows a diagram of the operation of a system to determinecompliance.

FIG. 12 shows the training and operation of a machine learning model fordetermining compliance.

FIG. 13 shows an embodiment for judging compliance of a lacrosse stickwith related game rules.

FIG. 14 shows a traffic stop where compliance with a “drunk driving”rule is being enforced.

FIG. 15 shows an office using a device to scan a suspected drive in thestop of FIG. 14.

FIG. 16 shows two suspected persons being tested in the traffic stop.

FIG. 17 is a diagram showing the training procedure for one level of atraining process of a machine model.

New Figures for the Current Application

FIG. 18 is a view of an embodiment using a camera with controlledillumination in a remote area.

FIG. 19 is a diagram of images produced by the camera of FIG. 18 withtwo levels of granularity.

FIG. 20 is a diagram of a sonic system to distinguish large objects in achannel.

FIG. 21 is a plan view of a system which uses a sonic coarse granularitysystem for detection and a camera for fine granularity.

FIG. 22 is a view of a system which used two levels of granularity inobservations from separate aerial vehicle.

FIG. 23 is a diagram of the functional steps of the system of FIG. 22.

FIG. 24 is a diagram of the developmental steps of the system of FIG.22.

FIG. 25 is a diagram of a portion of a simple recurrent neural net.

FIG. 26 is a diagram of the process for developing the pattern matching

software for a neural net.

FIG. 27 shows the steps and structure of the basic

information flow of a device and its training.

FIG. 28 shows the training and operation steps of the

model used to represent data to be evaluated and the situation forevaluation.

FIG. 29 describes an embodiment where anti-surveillance devices ae usedto detect an adversary device planted in an environment.

FIG. 30 is a diagram of the operation and development and operation ofthe system of FIG. 29.

DETAILED DESCRIPTION OF THE INVENTION Definitions

The definitions given in this section are intended to apply throughoutthe specification and in the claims.

Detailed Description from the PARENT and GRANDPARENT Applications

Detailed Description of the Invention of the GRANDPARENT Application.

This is also used by the PARENT application but is not new material. Thenew material for the PARENT application is below this material from theGRANDPARENT APPLICATION.

Escape Guiding Device

An escape guiding device, here usually called an escape device, is aportable device typically carried by a person entering a venue whichcollects information as the person enters and moves about the venue. Ifa threat occurs than the device provides guidance to allow the person toescape from the venue by a means or path that is calculated to allow thesafest and quickest egress.

Typical Use in an Emergency Situation

A typical use is when a person has carried the device into a venue andallowed the device to continuously gather information which is used totrain a computer learning system such as a neural net. When a threatsuch as a fire is detected by the device or the device is informed thata threat exist the device generates an escape plan with the model of thesituation developed in the learning system. The steps of the plan arecommunicated to the person who uses them to escape.

Use by Persons with Limited Abilities

The escape device would be especially useful when carried by a personwith special limitations on their abilities. For example, a person in awheelchair who may or may not have the assistance of a second person tohelp move the wheelchair, could have the device especially developing anescape plan which is suitable for wheelchairs. This may take intoaccount the restrictions or requirements on use of elevators during afire or panic. Persons with sensory limitations such as blindness anddeafness would also benefit from plans produced by the escape devicewhich take into account their situation in relation to the situation inthe venue.

The escape device when used by persons with limited abilities would inmany embodiments have specialized display transducers. This wouldinclude audio outputs; speakers; vibrators; large, bright or flashinglights; and tactile devices.

Another mode of use is by persons who use mobility assistance vehiclesor modern motorized wheelchairs. The device can communicate with thechair at several levels from using resources of the chair as a displayto complete autonomous control of the mobility device to carry theprotected person from the endangered venue.

Limitation that affect egress may be sensory, cognitive or related tomobility. Similar considerations may apply to persons who do not havelimited abilities but have responsibilities for other persons. Thisincludes caretakers for elderly persons and for infants or smallchildren. It would be necessary, for example, to make a special escapeplan for a person who has a very wide baby stroller with tripletinfants. Other factors would affect escape plans for devices where theprotected person is a child. A child may not be able to compete with apanicked crowd of adults in pushing for a narrow exit.

Professional Use by Rescuers and First Responders

There are several professions where persons as part of their jobs enterinto venues with high risks. These include firefighters and other firstresponders, the military in various operations, nuclear reactoremergency personnel and many

other kinds. They can operate in area where conditions change quicklyand information on current conditions may be hard to come by.

Venues

The escape device is envisioned as operating in an area called a venuein this specification and in the Claims. A venue here is an area whichis sufficiently large to allow the assumption that if a person escapesfrom the area then that person is safe from an anticipated threat. Themost common and typical venue is a building, but for some types ofthreats the venue may be limited to the portion of a building thataccommodates a specific activity or some other area which containshazardous zone. The extent of the venue is the area where the actual oranticipated threat is potentially operative.

Outdoor Venues

Some threats occur in an outdoor area. For example, a forest fire is athreat that is serious to people in certain situations such as to peoplefighting that fire.

The venue in this case is the area that is threatened by the fire. Otheroutdoor venues include dangerous neighborhoods, battlefields, and floodplains. The venue is defined by the range and nature of the threat.Another example is in a situation where a car has broken down in alimited access highway intersection. This is an area dangerous topedestrians and an the venue extends until an area safe for pedestriansis reached.

Special Venues

In one series of embodiments an escape device can be used to escape fromthe cabin of a passenger aircraft after that aircraft has crashed. Onentry to the aircraft the device determines the seat that the protectedperson is occupying and consults a contained or downloaded database ofaircraft layouts to determine the best escape path. Accelerometer andgyroscope sensors can evaluate the nature of the crash and with cluesfrom a camera a reasonable escape path and procedure can be generated.

Another important type of venue is an area where industrial or othertechnical operations is being performed. When things are going wrongescape from a chemical plant, a reactor site, a ship, or a crime scenemay be in order. An escape guiding device is useful in such situation,especially if the protected person is not familiar with the specificlocation or type of venue.

Threats.

The types of threats that may occasion the need to escape from a venuevary with the nature of the venue; but there are many different kinds ineach class of venue that may require such action.

The most well-known threat is fire. Fire can occur in buildings and trappeople by blocking exits and creating situations where exit by thestandard route or the entry route is impossible or inadvisable. Firecreates a lot of physical clues such as sounds, heat and smoke; but itcan also be hidden. Subtle use of sensors and information from serversmay be necessary to assess the extent of danger and to create a workableplan to escape the threat or the venue.

With respect to buildings, explosion and collapse are related to firebecause the original cause may be a fire. These events are usuallysudden, and an escape device is faced with the problem of evaluatingdata to determine what happened, where it happened and what limitationshave been created for an escape plan. In such situations, it is usefulto have access to data from servers concerning the details of the venue.Such data may be currently accessed or accessed prior to entry to thevenue. It may be used to pre-train the computer learning model allowingadditional training from new data concerning the threat event to allowgeneration of an escape plan.

Wildfire which is commonly referred to as forest fire in certain casesalso provides a threat that requires a sophisticated escape. There havebeen several cases where firefighters in rugged terrain have beensuddenly overtaken by flames sometimes with tragic outcomes. In manycases, local sensors would be unable to detect the threat coming; butwith server input concerning the threat a computer learning system cancombine that information with local sensor data concerning currentlocation and conditions observed in the venue to generate a reasonablepath. Such escapes sometimes need to be very quick and continuouscomputer monitoring of the situation with continuous training of a modelwould often be needed.

Terrorism and the related possibility of attach by hostile persons forother reasons has become a wide concern. Local sensor data concerningsuch events as sounds of gunfire, crowd noises and audible instructionsfrom venue staff or other persons can be combined with venue layout andlocation information in forming a plan.

The listed threats, other causes, and events caused by the dynamics ofotherwise unthreatening situations can cause panic. Panic itself can bea danger and threat that exceeds the importance of its original cause.Stampeding crowds have caused many injuries and fatalities. An escapedevice should in its basic levels of training take into account thesepossibilities and design escape plans which avoid particular crowddangers. These include narrow restrictions and cul-de-sacs whichthreaten a protected person with being suffocated or crushed by crowdpressure.

The threats listed above contain obvious serious dangers to persons; butthere are other threats which seem less serious but in certain cases maybe important. Nursing home residents, prisoners and other persons withlimited abilities to help themselves can be seriously threatened withloss of resources. These threats include loss of power and light, beinglocked in, loss of elevator service as well as other problems. Even whenstaff or other personnel are available to help with the situation, thenature of the difficulty or the number of persons threatened may beoverwhelming. For example, elevators and building lighting are in mostcases backed up by local generators in the event of a power supplyoutage. Unfortunately, generators often disclose their inoperabilityonly after they are needed. An escape from a high floor in an unlightedstairwell is very difficult. Even if most threatened persons can“shelter in place” it may be necessary to send someone for help orneeded supplies.

Escape.

Escape from a venue under threat means to move away from the venue to aplace where the threat is no longer a danger. Escape from a threat in avenue means to either move away from the operative area of the threat orto take action which renders the threat inoperative. The person carryingthe escape device could be a leader of a group and could be charged withleading the entire group to safety.

A key function of an escape device is to form an escape plan. An escapeplan can include a path to travel, conditions to be check for whichwould modify the plan, and conditional actions in the event ofdetermination of necessary conditions. An example plan could beexpressed in words as “travel back to the entrance, but if it's blockedby a crowd go to the side door.” The sensors in the device inform themodel which would in effect watch for signs of a crowd and redirectmovement to the side door if they are found. The plan may contain routesor paths to escape from the specific venue which comprise locations anddirections to facilitate movement away from the hazard or venue.

While the entire plan could be made available to a person using thedevice, in most cases the device would display or convey to the user thenext step to take in escape. Keeping the presentation simple may be veryimportant in a sudden threat situation.

Sensors

The escape guiding device can employ information from a variety of typesof sensors. These include video or picture cameras, accelerometers andgyroscopes, GPS receivers, receivers for transmissions from remotesensors, threat sensors not a part of the device. The term sensor asused in this specification and in the claims, includes not only sensorsthat are part of the device but also receivers that collect currentlysensed information and are provided by the venue or third parties. Italso includes a GPS receiver where the receiver generates locationinformation from timing details of the received GPS signals.

Cameras

A camera either still or video can gather information. Machine visionsoftware is now available which will recognize situations and objectsand can provide a critical input to a computer learning system. Suchlearning systems often work in several layers and provide learning at ahigh level in a manner using generic detail recognizing systems at alower level.

Accelerometers and Gyros can be used to detect movements of the deviceor the person carrying the device. In the phase prior to detection of athreat, such information can be correlated with venue layout, GPS andother information to give a more detailed picture of the entry path.Slowdowns, halts, and diversions on entry may or may not be clues topotential difficulties on egress. That may be determined by correlationwith other information. In the escape phase after a threat is detectedother movement events are important information. Slowdown, halts anddiversions as even more important here. Other events such as a personfalling or being knocked down can be detected and taken into account.The rate of movement and details such whether the movement of the personescaping are important

Microphones

Microphones can be used to detect conditions around the escape guidingdevice in several ways. If multiple microphones are in use and havedirectional sensitivity that can be correlated with directionalinformation from sensors.

One category of useful sounds are those made by other persons in thevenue can give important clues to the movements of persons, to thenumber of persons in various area and the experiences that persons arehaving. It may be possible by using analysis of the sounds of persons todetect potential escape routes which are blocked or where panics areoccurring.

Another category of useful sounds are those made by the threat itself.Fires and gunshots as well as the panicked crowds mentioned above havecharacteristic sounds which are subject to analysis.

A third category of useful sounds are announcements and otherinformation given by the venue operators. Sensitive microphones andanalytic software can interpret these sounds where they would be coveredup by other ambient sounds as they are heard by persons.

Miscellaneous Sensors

Other types of sensors which may be useful include magnetometers whichcan give an absolute direction from the earth's magnetic field incertain cases and thermometers which can detect heat from threats andchanges of temperature from outside area in cold weather.

Output Device.

The term “output device” is used in this specification and in the claimsrather than simply “display” to convey that the output of warning andguidance for the protected person may be in modes other than the commonvisual screen.

Especially with persons of limited abilities and for most persons indifficult environments, a visual screen may not be the best way toconvey the needed information in a way to get timely escape action. Someoutput devices work by conveying information to a person and some byconveying information to equipment such as an autonomous vehicle whichacts on an escape plan.

Some output devices are visual. They can be general display screenswhich can show pictures and text and provide detailed instructions.These can be integrated with input devices such as touch screens. Othervisual output devices include lights which can be flashing to getattention and mechanical devices which raise flags.

Some output devices are audible. This includes speakers and earphones aswell as sirens which may be operated over a communication link.

Tactile output devices include vibrators, braille devices andspecialized devices which operate mechanical signifiers.

The escape device can operate vehicles as the primary or an additionaloutput mode. In other cases, the escape device can operate elevators,doors, open gates and substitute for a key or pass code to allow exit.

Beacons

Beacons can be placed in the device by the venue operator. The beaconscan operate by radio, infrared or other communication means. An ordinaryExit sign is in effect a beacon. Such signs may incorporate beacons thatoperate in other modes for use with escape devices. More sophisticatedvenues may come to offer services with venue information at theirlocations. In the future, digital escape information may come to berequired as an extension of the services for direct communication tohumans. Current flashing fire warnings, sirens, buzzers, exit signs etc.may have digital equivalents and augmentation. An intelligent escapedevice would take advantage of such information as much as possible.

Processor, Machine Learning and Models

A device to use gathered information to guide a person from a venuewhere a serious hazard exists requires very sophisticated computersystem to complex problems in interpreting incoming data and generatinga plan for escape. In general, it is not feasible to discover all of therules and relationships necessary to solve that problem and to write adeterminative computer program that produces a sufficient result.However, methods have been developed and are wide and increasing to usea set of examples which is processed and used to product a set of ruleswhich working together can find answers suggested by the examples. Asubstantial set of examples and a large amount of processing arerequired but many people are trained and are being trained inapplication of well known methods to implement this approach on a widevariety of problems.

There are limitations on the kind of problems that can be solved withthis approach, but the problem here is well suited for the approachbecause of the specific input data that is to be gathered and because ofthe specific kind of output that is required.

Machine learning as used here and in the Claims is a term for the typeof artificial intelligence that is implemented without explicitprogramming to generate solutions to the problems confronted. It isfocused on the development of solutions by having the effective programchange as it incorporates data in such a way that it converges on anability to produce the required solution.

Model

The term model as used in this specification includes representationsthat do not explicitly describe the system modeled but which aredesigned or trained to produce information based on the behavior of thesystem. It includes pattern matching system such as a neural networkwhich has been trained on data exemplifying the system. In that case themodel consists of a, probably huge, array of parameters that determinethe operation of the individual neurons in the neural

net program. Training would work by systematically adjusting the valuesof these parameters on the basis of the training data.

Detailed Description of the Invention of the PARENT Application.

Standards

A standard as used in this specification and claims is a specificationof a condition on which the granting of a privilege is conditioned. Thestandard can be expressed in words directly defining it or in aprocedure which results in a decision determining whether or not thecondition warrants the granting of the privilege.

The procedure can be defined in words or by means of a device whichevaluates the condition.

Physical Standards

A “physical standard” as used in this specification and claims is astandard which depends on the existence of physical attributes which canbe evaluated, possibly in the context of non-physical attributes, todetermine compliance with the physical standard. Examples include thepresence of a necktie or other class of object, size of an object, andthe presence of certain text. Physical standards are not necessarilybased on static attributes but can be based on upon actions produced byphysical objects or physical actions of persons. These actions may beconditional on certain stimulations or environments.

This specification disavows the full scope of the term “physicalstandard” by excluding physical standards which consist wholly orprimarily of identification of a specific person or object. Thislimitation is intended to invoke the decision of Scilled Life Sys., Inc.v. Advanced Cardiovascular Sys., Inc., 242 F.3d 1337, 1341 (Fed. Cir.2001) (“Where the specification makes clear that the invention does notinclude a particular feature, that feature is deemed to be outside thereach of the claims of the patent, even though the language of theclaims, read without reference to the specification, might be consideredbroad enough to encompass the feature in question.”) Therefore, aphysical standard that consists of requiring any person with certaincharacteristics or a license tag from a certain jurisdiction on avehicle be present would be included in the definition; but, a physicalstandard that requires only that a specific person or a specific licensetag number be present would not. A standard may comprise a portion thatconstitutes a physical standard in conjunction with a portion which isnon-physical or which is excluded above as a specific identification.

Compliance with Physical Standards

Because it is generally impossible to implement rules sufficientlydetailed to determine compliance with any set of rules in specificcases, a decision mechanism

is necessary to apply rules. If the decision mechanism is automatic andbased on a program containing a more extensive system of rules than thatthe published rules, it too is likely to have situations which are notdecidable. Such systems are common. Systems based on human judgementrange from the decisions made by “bouncers” at the door to nightclubs tothe U.S. Supreme Court. Unfortunately, these systems typically do notallow persons wanting to determine in advance what to have or do to bein compliance. Irreversible actions to achieve compliance may fail withthe failure only determined at the last minute. Systems based on humanjudgement are also subject to variation from decision to decision andfrom judge to judge due to unavoidable variations in human behavior.

Physical Attributes

Compliance with standards is determined on the basis of informationavailable to the decider. There are several distinct classes ofinformation used for this purpose including information from records,information testified to by a witness, declarations of fact from anapplicant, declarations of intentions and physical attributes of aperson or object relevant to the standard to be applied.

Physical attributes are those that are evaluated in a particular objector person at a specific time. For example, the current length of aperson's hair or beard is a physical attribute. A camera or video imageof a document is a physical attribute of that specific copy of thedocument, but the validity, uniqueness or meaning of the document is notin itself a physical attribute.

Testing of Compliance by Evaluation of Physical Attributes

In many cases standards can be expressed by requiring some state of aphysical attribute. For another example a nightclub can express itsstandard aimed at insuring that all customers are of legal age to buyalcohol by requiring that they have a governmental document with apicture certifying the age. This would be a document such as a driver'slicense. The physical evaluation to test compliance would be to comparethe picture to the person seeking entrance. Human comparison of pictureswith actual persons is very slow and unreliable. The comparing person islikely to be distracted by irrelevant characteristics such as hair styleand eyeglass designs. A substantial proportion of people have acondition, called prosopagnosia or face blindness which interferes withprocessing of visual data concerning faces. A person seeking admissionrisks embarrassing rejection at the same time the venue risks admittingimproper persons.

Another example can be seen in a requirement that a vehicle tag renewalsticker be attached in a specified manner, such a being right side upand in a specified place on the license tag. This could be tested by anapplication

downloaded on a user's personal device (cell phone) which would evaluatea picture of the installation, giving the user confidence that theinstallation is correct.

Machine Learning

Machine learning is a well-developed and understood method of creatingdevices that are capable of solving problems that defy development ofprogrammatic solutions that rely on understanding in detail the workingof the system to be analyzed. A famous example is the modern languagetranslation systems widely used on personal computing devices.Development of programs to translate languages has produced poor resultsbecause of the complex and subtle structure of human languages and thescale of the problem. But systems have been developed to be trained on alarge (possibly hundreds of millions) number of examples of languageusage. The trained models are then applied to an input in one languageand provide output which is very likely to be a satisfactory translationin another language of that input.

Machine learning systems are very different from computers runningprograms written to model problems to be solved. While theimplementation of a machine learning system may be made by means of acomputer program, this is not the only way to implement machine learningmodels. An array of analog devices (usually called gates) can implementthe model in a massively parallel way. Rather than containing a program,a machine learning system constructs a model which transforms an inputthrough a huge number of gates to produce an output which has astatistical meaning. The operation of the gates is modified in thetraining steps until the behavior of the model converges on a tendencyto produce desired results.

Machine Learning System Models

A machine learning system model or just “model” as used in thisspecification and in the claims is a large set of parameters representedas data or

physically and arranged in such a way that they can be adjusted by atraining process based on a collection of data that represents thesystem being modeled. The model also allows inputs that represent aparticular state or set of states of the system to be analyzed by use ofthe model. The use of the model transforms the inputs into a set ofoutputs that constitute an analysis of the states being analyzed.

A model can be applied to a set inputs by means of an algorithm executedby a processor or by means of a physical analog device to perform thetransformation. The algorithm or device is only the means of evaluationand is distinct from the model which is the set of trained parametersand the structure in which they interact.

Model of an Area of Application of a Standard

The model being trained can be trained using data from examples whichare demonstrative of compliant and non-compliant situations, objects andbehaviors which are more general in scope that the particular case to beexamined in a final evaluation of the model. This produces a model of anarea of application of a standard more general than one which is trainedto a specific context. This would typically be done on larger processerswithout real time constraints prior to downloading the model to a devicefor use in specific contexts.

Training a Model

In this specification and in the claims the process of training a modelconsists of applying data representing possible inputs to the machinelearning system with the model in its current state of possibly partialtraining. The outputs of the system are used to generate incrementaladjustments to improve the transformation of the inputs into outputsbetter representing the desired behavior of the system.

The usual way to determine the adjustment to be made to the model foreach group of inputs presented is to calculate or measure the effect onthe outputs of each parameter in application of that set of inputs. Ifthe effect is favorable in providing outputs that correspond as wantedto the inputs then the parameter is very slightly augmented to improvethe overall behavior of the model as trained.

There are many ways to accumulate the data sets used for training. Oneway is to find or set up a large number of examples with known outcomesand collect the data from them. Another way is to write an algorithmwhich generates examples. The examples can be graded by people or thegeneration method may be able to predict the outcomes. Some problems areeasy to solve in reverse; i.e. a set of inputs may be easier to get froma assumed output than to find the output from a set of inputs. Forexample, to train a system to distinguish pictures of dogs from picturesof cats one can get pictures from public sources such as the internetand use humans to label the species depicted. That set can be used totrain a model which can test other pictures.

Convergence

The training process is continued for each item in the training setdata. Because it is important that training result in a stable andgradual progression of the model toward the desired behavior teach roundwhich uses the set of training data items only changes the model by asmall increment. The rounds are repeated many times and the results arecompared to data reserved for testing in order to measure theeffectiveness of training. If the structure of the model is well chosenthan parameters will converge on values that produce the desired outputsfor various input sets.

Training in Levels

Models can be arranged in levels both for training and for evaluation ofinputs. The application of the model to a set of inputs generatesoutputs that

describe in a higher level of generality the meaning of the inputs.Those outputs can become inputs to further structure which is a modelfor a more general transformation of the original inputs towardmeaningful outputs.

In this specification and in the claims, a level of training is thetraining of a portion of the parameters of a model to produce outputsthat are trained until a state of convergence is attained and madeavailable for input the next portion of the model. That is, distinctlevels are made distinct by separate training to convergence. It ispossible to simultaneously train multiple levels, but they are distinctlevels when they are separately tested for convergence. A level that isnot tested for convergence but which uses inputs from a level that hasbeen brought to convergence is a distinct level from the level providingthe inputs.

Typical models are in at least four levels. The first which here iscalled the Basic level takes raw sensor input and describes it in termsdirectly definable based on the input data. Examples would be detectionof edges from visual data and of tones, harmonics and burst timings foraudible data. The second level which is here called the General level isto identify objects and events from the output of the first level.Examples would be to detect a person crossing the path of the sensor oridentifying a sound as a gunshot or crowd noise. The third level, hereincalled the Specific level is to allow the model to identify actions andobjects appropriate to the purpose of use of the model. Examples of thislevel include model layers to implement steering or acceleration of avehicle or determination of compliance with a standard in a specifictype of situation. There is also a fourth level called the In-Use levelin many implementations. This level incorporates data collected while amodel is in use which modifies the model to allow evaluations at a latertime to take into account earlier inputs or evaluations where a seriesof evaluations is made.

Implementation of Training on a Processor with a Memory

Training requires a very large amount of processing to apply the largeamount of data in the training set repeatedly to incrementally cause themodel to

converge on the desired behavior. If the adjustments from one passthrough the data are too large, then the model may not converge or maynot allow the effects of all of the inputs to diffuse through the modelstructure and correctly operate. For this reason, specialized verypowerful processors are used for training. They are not appropriate forincorporation in portable devices because of considerations of size andexpense.

Basic Training

In this specification and in the claims, basic training refers totraining which is used to interpret inputs from sensors or raw data fromdata sources to identify aspects of objects and actions treated asobjects that are implied by the data and too general in nature toidentify the potentially detected objects at this stage.

Examples include edge detection, categorization of sounds by location ofthe source, face detection, orientation in space, counting objects,elimination of backgrounds and many other general tasks ofinterpretation.

A portion of a machine learning model with this training can be used formany applications and could be supplied by a specialized developer. It'straining would be brought to convergence and the outputs supplied to thenext level of training when the model is used to evaluate inputs eitherfor further training of other levels or in actual use.

Data for General Training Describing the Area of Application of a Model

Data for the general level of training can be acquired by collecting anumber of real examples or by generating examples with a program toproduce examples and training data. In this and in other levels, it isoften much easier to produce a program for generating examples formachine learning than to determine specific rules to allow determinativenon-learning algorithms for evaluating rules designed for humanunderstanding.

Data for Training Concerning Compliance with a Standard

There is often available a set of examples to be made into training datafrom prior application of a rule set. For example, a dress code modelcould be made by using video collected over a period of time of peopleentering a venue. The videos could be graded by whether persons areadmitted or turned away by entry personnel. This would allow automaticgeneration of a training set by processing the videos.

Transferring a Trained Model

Levels of training of a machine learning model can be divided into twoclasses. The first class is those levels that require very large amountsof processing power and time to train. These typically use largetraining sets and are done before other levels. They include in mostcases the basic training levels which are concerned into extractinginteresting features from raw data usually from sensors and the generaltraining levels which concern coordination of features in sets ofcircumstances which are designed to encompass the specific situation tobe evaluated. These levels cannot be conveniently handled in real timeand on a processor convenient to take into the field to performevaluations.

The second class of levels are those that must be performed after thespecific situation to be evaluated is determined. They must be performedin real time and on processors available at that time. The model trainedby the first class of levels can be transferred to a more convenientprocessor for the second class of levels of training.

Data for Testing Compliance with a Standard

After a model is trained in several levels and downloaded to a processorto use the model to evaluate situations, data must be collected with anappropriate sensor. The data is provided to the processer as input tothe model for an

evaluation to produce outputs. The outputs may have further non-machinelearning processing to produce a determination of compliance with thetesting physical standard.

Acquisition of Testing Data with a Sensor

A portion of the data collected or generated at each level is reservedfor testing. This data is not used for training to enable testing themodel without concern that the model might be effect only with thespecific cases used for training.

Determination of Compliance

Once the model is trained through all relevant levels, it is used toevaluate a situation to be tested for the state of compliance. Data iscollected by use of sensors, servers or other sources and supplied as aninput to the machine learning model. Processing the input through theprocess defined by the model generates outputs. These outputs are usedto determine compliance.

Detailed Description of the Several Views of the Drawing

Drawings from the GRANDPARENT Application.

Referring to FIG. 1:

A person 10 to be protected is entering into a venue at an entrance 11

carrying an escape device 12. The device gathers information as thevenue is entered and the person progresses through the venue along path13. The device gathers information from sensors which may include anoptical camera(s) with a field of view 14 and self contained sensorssuch as an accelerometer or gyroscope.A beacon(s) 15 in the venue may provide additional information to thedevice with a suitable signal 16. The device both before and afterentering the venue may receivesignals 17 from the outside servers which provide various kinds ofinformation such as location (gps), venue layout, pre-calculated pathsand parts of paths and many other kinds of useful data. All or portionsof the calculation burden may be offloaded to external servers. Theprotected person proceed along the path which may contain variouspossible areas 18 where on return false turns may be taken or additionaldangers may exist. The inner parts of the venue are the destination andthe activities for which the person has come take place 19.

Referring to FIG. 2:

The person 10 from FIG. 1 is in the inner part of the venue with theescape device 12 when a threat 20 appears. The threat depicted is a firewhich has broken out. A large number of other people have entered thevenue. If the venue is full of panicked people as depicted (numbers areomitted for clarity) and the lighting has failed, efficient escape maybe difficult. In many cases the best escape is not back through theentrance path, but in the depicted case that path has been chosen by thedevice and is shown as 13. Two cull-de-sac's 18 are shown. They may nothave been noticed by the protected person on the way in; but they areeach holding a crowd of people who think that is the way out. Withoutthe help of the escape device, the protected person may choose to jointhe people trying unsuccessfully to force an impossible exit.

Referring to FIG. 3:

A front view of one embodiment of an escape device is shown. Thedepicted device is designed to be worn on a chain, ribbon or lanyardaround the neck of a protected person with a provided eye 30 forattachment. The central area 31 of the device is a display screen whichgives instructions in case of a detected hazard and shows a decorativescreensaver chosen by the user at other times. A camera 32 is used togather information on entry to the venue as the device hangs around theprotected person's neck. Eight sound ports 33 lead to microphones whichgather audio information to be analyzed for content, timing and arrivaldirection. A fringe

34 around the device is for decoration and to help distinguish the twosides of the device. As depicted the device is showing the detection ofa threat 35 and is beginning the process of guiding the protected personfrom the venue with exit directional information 36. The screensaverdecorative display has been replaced with a warning.

Referring to FIG. 4:

The back of the device is shown. It is cut away as necessary to showinterior components. A speaker 40 is included to alert the user. Variousinput and sensors are shown including an accelerometer 41, a gyroscope42, a GPS receiver 43 and a communication device 44. The communicationdevice can be used for many purposes such as receiving beacons from thevenue, information about the venue from an outside server, access tooutside processing assistance from a server or download of pre-trainedparts of a pattern recognition or neural network for the device. Use ofthe outside servers allows function with a reasonable processing load onthe devices own processor 45.

Referring to FIG. 5:

A venue is depicted where the escape device is adapted for escape froman aircraft cabin 50. The protected person enters from the normal cabindoor 51 along path 52. The protected person arrives at assigned seat 53.The device integrates information downloaded from servers such as thepersons assigned seat, the layout of the particular aircraft for thisflight, the anticipated number of passengers and the scheduled times ofthe flight. This information is used in combination with informationgathered by the sensors as the protected person enters the aircraft andafter a hazard is detected. If a hazard situation is detected the escapedevice evaluates all available information and plots an escape route. Inthe instant situation, the escape route 54 is not toward the entrancedoor but is in the other direction to the exit aisle 56 which leads toan emergency exit 56.

Referring to FIG. 6:

The inclusion of an escape device in the equipment of persons in variousroles is shown. The device is designed to be able to access servers andthe environment with sensors and communication device, perform it'sanalytical role and provide escape information to the user. It may alsobe integrated into other equipment associated with the protected personsrole both for purposes of gathering information and for outputtingescape paths.

One situation where integrating an escape device into a person's dailyequipment is in the case of a firefighter 60. The device could havesensors 61 and a display and warning device 62. In this case it would behelpful to provide specialized sensors such as infrared heat detectors.If the firefighter comes to a closed door with intense fire on the otherside, it is dangerous to open that door.

Firefighters are so trained, but forgetting this critical rule is asource of many injuries. If the firefighter is attempting to escapeother critical dangers, the chance of making such a mistake ismultiplied. Heat sensors in the escape device could warn the firefighternot to take that route and the device could suggest another escape plan.

Another situation is that of a military person 63 or soldier. Manybattlefield conditions can affect the choice of a suitable escape routefrom a dangerous area. Directional sound detectors in a sensor module 64would be useful in detecting area with friendly or hostile occupants oractive use of weapons.

Taking these factors into account a computer learning system would beable to estimate an appropriate route for accessing a safer area such asin returning to ones unit. Directions could be displayed in a smalldisplay unit using an appropriate sensory mode 65.

A forest firefighter is shown at 66. In this case the protected personis using a water pack to put out a small fire separated from the mainarea of threat. Heat sensors as part of a sensor package 67 would beuseful to detect the approach of active burning and may save a person indanger of being surprised. In particular, continuous planning of anescape route can cause a danger of being cutoff to be detected beforethe actual threat approaches. Again, a display and warning device isused 68.

A mobility chair or motorized wheelchair is shown at 69. The sensor unit70 gathers information from servers and sensors. In the event of athreat after a plan is developed appropriate instructions are display ona display 71. In some cases outputs from the escape device are used todirect the chair controls to make the escape or facilitate the neededmovements. For example, the chair may have built in limitations of speedand other parameters that can be overridden.

Referring to Fig. T

An embodiment of an escape device in a smart phone 80 is shown. Typicalsmartphones have many relevant sensors 81, substantial processing powerand visual, audible and vibratory output devices 82. With suitablesoftware embodying a learning model of venues an escape device can beimplemented as an application in a smartphone.

Referring to FIG. 8:

An embodiment of an escape device integrated into a vehicle 90 is shown.The escape device uses a sensor and communications module 91 mounted ontop of the vehicle. When a threat is detected the plan developed by theescape device is displayed on the GPS display 92 integrated into thevehicle. The input devices for the vehicle GPS 93 are also used to enterparameters for the escape module. In this way the vehicle user caninstruct the escape device about servers and sources of information tobe considered.

Referring to Fig. g:

The steps and structure of the basic information flow of the device andits training are shown. There is a timeline 100 which is divided intothree periods.

First pre-entry 101 which is the time prior to use of the escape deviceto track entry and movements into a venue. The device is pre-trainedduring this time. The next time period is pre-threat 102, the time priorto detection of a threat which is used to train the escape device andstore relevant conclusions. The final time period is the threat andescape period 103. During this time collection of data and trainingcontinue but evaluation of the model and development of escape plans andinstructions also occurs. The instructions are put to the output displaytransducers to allow the protected person to escape the venue.

During the pre-entry period, the model comprised in the escape device isconstructed and initialized to parameter values suitable to alloweffective training

104. Pre-training 105 typically proceeds in two stages and uses two datasets. The first stage uses a generic data set and trains for generalskills such as recognizing objects and edges from a camera sensor. Thedata set for this stage may not be one specifically developed for thisapplication but is suitable to the sensors and processing resources tobe used in the device. The second stage used a data set developed 105specifically for this application which is based on real or virtualescape and venue analysis scenarios. Such data sets would find dataconcerning specific venue layouts and facilities and data concerningtypical venues that can predict possible situations in actual venues tobe especially relevant. The data sets are trained 106, usuallysequentially, into the model. Low level training. such as here used, iscomputation intensive; but it can be accomplished on a one-time basisusing substantial computing resources not needed by the final escapedevice. If the device is produced in mass quantities the pretrainingresults are loaded into eachdevice in the process of production. Additional data sets and trainingcan be done prior to use of the device to adapt it to specificanticipated uses or venues.

During the pre-threat period, the protected person takes the escapedevice into the venue. Sensors in the device collect 107 informationabout the venue and the protected person's location and progress.Information is also downloaded 108 from various servers and beaconsprovided by the venue or outside sources. The sensor and serverinformation is used to do more venue specific training of the model.This can be accomplished by a processor in the escape device; but it mayalso be assisted by training resources located in servers away from theescape device. Sensor and server data can also be stored 110 to be usedas an input in the later threat stage. The escape device is then readiedto receive a threat.

During the threat period, the escape device has been told or hasdetermined that a threat is detected and escape plans and actions shouldbe undertaken. The model continues to be trained 111 from allinformation sources including stored data 112 and data that continues tobe collected from sensors 113 and servers 114. The model is evaluated115 based on its current training 111, stored data 112, continuingsensor data 113 and continuing server data 114. Outputs to guide theprotected person are generated 116 and presented for action.

Referring to FIG. 10:

A diagram of the process for developing the pattern matching softwarefor the embodiment of FIG. 9 is shown. The software to be developed is arecurrent neural net with an input layer, a recurrent layer, aadditional fully connected hidden layer and an output layer. Other morecomplex structures are also easily implemented but this structure wouldbe sufficient for the low level of accuracy needed in this embodiment.

The first step of the development is to accumulate 120 a data set fortraining and testing. Several data sets are required as described forFIG. 9. One set

of data is collected by a device that is constructed similarly to thatof FIG. 3 and the processor is programmed to collect data from thesensors and transmit it via the communication module to an outside datacollection system. A substantial number of escape scenarios enacted orsimulated and the resulting data is manually graded and entered into thedatabase.

The data is divided into two sets with a larger portion for training anda smaller portion for accuracy testing. This is considered labeled databecause it contains both input (sensor) data and the desired output forthat data (presence or absence of transfer).

In this and related embodiments, a step in the development which mightbe started in parallel with data collection is the design of anappropriate neural network. The sizing of the layers and the setting ofvarious factors in the neural net which are in addition to the factorsand values (parameters) that are adjusted in training are collectivelyreferred to as hyperparameters to distinguish them from the “parameters”which are adjusted in training the neural network. The hyperparametersare initialized 121 to appropriate values. In some systems that aretaught hyperparameters are adjusted during the course of training butare distinct from trainable parameters because the adjustments are onthe basis of the progress of the training rather than being directfunctions of the data.

The next step is to initialize 122 the parameters which are to betrained. Appropriate initialization is necessary for reasonably rapidconvergence of the neural net. A number of techniques are taught toproduct an initial set of values which produced good training progress.

The network is then trained 123 by passing data set items through thenetwork as implemented on a training processor. Because trainingrequires larger processing power and time than use of the network aftertraining special powerful processors are used for this step. Thetraining process adjusts the parameters incrementally on the basis ofthe output of the neural network. The

hyperparameters specify the methods of calculating the adjustment toparameters. Generally, the output of the network is used to backpropagate through the network to provide further input to theadjustments. The items in the training portion of the dataset are usedrepeatedly while the convergence of the network is observed 124 byprocesses in the training data processor.

If the convergence is judged 125 not to be adequate the training isstopped, the hyperparameters are adjusted 126, the neural network isreinitialized and the training process is repeated until satisfactoryconvergence is obtained. The smaller portion of the data set which hasbeen retained and not used for training is then passed 127 through theneural network (classified) and the output is checked 128 for accuracy.If accuracy is not sufficient for the goals of the particular systembeing developed then the net structure is made larger 129 and thetraining process is repeated until satisfactory accuracy is obtained.

The trained neural network is then downloaded 130 to the target device,which is then ready for system testing 131.

Referring to FIG. 11:

The basic operation of a compliance system is shown. In the depictedembodiment both acceptable 210 and unacceptable 211 images are used totrain 212 a computer learning system model 213, which embodies thedesired compliance. If the user is a night club venue with a dress codethe images concentrate on clothes and other characteristics covered bythe dress code. The model is uploaded 214 to a website 215 which isaccessed by potential customers of the club. The model is downloaded 216to an application in the customer's smartphone 217. The application usesthe smart phone's camera to view 218 the customer or the customersproposed garments 219. The application has a computer learning moduleappropriate to apply the downloaded model to the image of the proposedgarment and displays a decision 220 concerning the acceptability of thegarment.

Referring to FIG. 12:

The training and operation steps of the model used to represent rules tobe complied with and the situation for compliance is shown. This diagramis intended to show the relationships between different levels oftraining of a model each of which builds on the levels below. The flowof the training and application of the model are shown by the arrow 230.The main diagram of the figure shows the detailed steps. The descriptionis for a general neural net type of model. Other model types can followthe same general flow and neural nets can incorporate implementationdetails not shown. The model generally has layers which are arranged inthe same order as the training steps and when a particular level of themodel is being trained, data is evaluated by the earlier levels of themodel which have already had training to provide inputs to the levelbeing trained.

The model is initialized 231 with suitable values in a trainableparameter set. A basic data set 232 with basic information is used toperform the first level of training 233 the model. The model wouldgenerally have multiple layers and the basic data set would be used totrain the earliest layers of the model. It would use data to allow theselayers to recognize or react to features such as edges in pictorial dataand sound impulses for audio data. This training would be applicable tomany applications of a machine learning system. It may be provided by asupplier of implementation and hardware systems and these layers may beacquired in an already trained condition by implementations ofapplications. In the embodiment of FIG. 11 showing a dress code standardthe first level would be similar to most models and be designed toprocess data from the kinds of sensors in the device that will finallyuse the model to evaluate compliance with the standard for the dresscode.

The second level of training 234 in the depicted embodiment is done witha second “General” data set 235. This data is selected to allow themodel to use inputs to recognize objects and entities relevant to theapplication of the model.

The general data set in the depicted embodiment is generated by acombination of data generation from a simulation 236 of generalapplications of the model and specific data gathered 237 for suchapplications. The applications at this level include recognition ofobjects such as persons, articles of clothing, signs and other itemsused to define and the movements made by sensors as they traverse ascene. Prior to the training at this level layers are typically added238 to the model to allow the training to take effect in facilitatinganalysis with the aid of the moved based on inputs processed bypreceding levels of trained model.In the embodiment depicted in FIG. 11 the second level would be able torecognize kinds of clothing, determine the content of text, and find outother factors that would serve to distinguish suitable and non-suitabledress. It would likely be generic for implementation of dress codes ingeneral but would not at this level be using the specific dress code tobe enforced.

The third level of training 239 in the depicted embodiment is done witha second “Specific” data set 240. This data is selected to allow themodel to use inputs to recognize objects and entities relevant in acontext relevant to a particular application of the model. The generaldata set in the depicted embodiment is generated by a combination ofdata generation from a simulation 41 of specific applications of themodel and specific data gathered 42 for such applications.

Typical information used to generate a simulation at this level includemany variations of relevant objects for the purpose of applyingstandards similar to the one to be implemented. Prior to the training atthis level layers are typically added 243 to the model to allow thetraining to take effect in facilitating analysis with the aid of themodel based on inputs processed by preceding levels of trained model. Inthe embodiment of FIG. 11 this level is used to train the model on thespecifics of a dress code standard. Pictures of acceptable andnon-acceptable dress are used for training to set up to allow the modelto predict the acceptability of test sets.

After the model is trained through several levels, it is usuallydownloaded 244 from high powered training processors which are only usedto prepare the

model to a smaller portable processor to execute the model in actualuse. To use the model to evaluate a situation data is gathered from thesituation 245 by means of appropriate sensors and prepared to serve asan input 246 for the model. The model on the basis of (evaluating) theinputs generates 247 outputs 248 corresponding to the action of thetraining on the parameters of the model.

In some more advanced implementations of the system, inputs and outputsare used to select 249 additional training for the model. Theinformation in the inputs and outputs can cause the download of sets ofparameters which can be added to the model or a limited training processsimilar to that used to develop the original model can be accomplishedby the evaluation processor.

Referring to FIG. 13:

FIG. 13 shows an embodiment where a machine learning compliance systemis used to check women's lacrosse sticks for legality. The usual waythat female lacrosse players take the ball from opponents is to dislodgethe ball by striking the opponents stick with their own stick. A stickwith a pocket that held the ball too tightly would be an unfairadvantage. A complex system of rules for lacrosse sticks has grown up toprevent this. The particular existing rules used to illustrate thisproblem her are those published by the Federation of InternationalLacrosse, Women's Sector. During games the officials perform inspectionsof sticks which usually only check if the pocket has been made too deepby failing the expose the ball above the siderail of the stick. Theparticular rule is “The top of the ball, when dropped into the pocket ofa horizontally held crosse, must be visible/seen above the top of theentire wooden or plastic sidewall after reasonable force with one handhas been applied to and released from a ball.” This rule is fairlyobjective and can be reasonably applied by an official on the field, butthere are many other rules which require judgement, measurement withinstruments, and detailed interpretation to apply. In practice, theenforcement of these rules is by a system of approval of samples of newmodels of lacrosse sticks submitted by manufacturers.

The approval system greatly slows adoption of new models and does notaddress variation after manufacture from wear, repair, user stringingadjustment, user modifications and outright cheating.

A device with a downloaded machine learning model and sensors canperform a much more thorough check of a lacrosse stick and provideconfidence that the rules are being observed.

Referring again to said FIG. 13, a lacrosse stick 270 is shown beingchecked by an official 271 without the help of any device. The officialobserves that the ball 272 placed into the pocket extends higher thanthe sidewall of the stick's head. This stick passes the inspection.Another stick 273 is shown with a ball that does not extend over thesidewall and is declared illegal. This is usually the only checkcurrently done, but an official may or may not observe other violationsand will declare other faults in a potentially inconsistent manner.

A lacrosse stick 274 is shown with its handle end placed on a flatsurface 275 and with a device 276 containing a sensor 277 placed on thehead of the stick. A beam 278 from the sensor, which could be light,infrared, sonic or radio in nature depending on the choice of sensor, isprojected to the flat surface and the length of the stick is measuredfrom the reflection. The length determined by this process is used asone input to the machine learning model implemented in the device. Bywell know principles in the field of machine vision, various otherdimensions of the object can be determined from use of a camera 280 orvideo sensor in a device 279, which may be the same device used tomeasure the length. These additional measurements are used as additionalinputs to the machine learning model. The case depicted the visionenabled device is requesting that the lacrosse stick be rotated toprovide additional angles for visual inspection. There are severalmeasurement based rules such the requirement that the overall length ofthe head be between 25.4 cm and 30.5 cm.

There are many rules that could be enforced by a machine learning systemother than the linear measurements above. An additional example is shownin the laced lacrosse stick head 281 shown with three of its laces 282run through a slot 283 above the lacing holes. This is in violation of arule concerning attachment of a pocket to the head which requiresattachment to the bottom of the bottom rail. This rule is reasonablebecause the variation in attachment heights of laces could provide atrapping effect for the ball which would be an unfair advantage. Therule as stated in the rulebook is complex and requires pictures to guidethe officials.

Violations in practice could be easily overlooked and an automatedsystem would be helpful in preventing them.

Referring to FIGS. 14 to 16:

Law enforcement officers are often required to make judgements whenmotorists suspected of driving under the influence of alcohol or otherintoxicating substances. Chemical tests for alcohol levels are intrusiveand in some places can only be required on the basis of a specific levelof suspicion which may later be overturned in court. A non intrusivemethod of screening would be useful in eliminating the need for chemicaltests in some cases and in justifying the requirement in other cases. Adevice with an appropriately trained machine learning model which videoand audio information taken or recorded can provided this function.

FIG. 14 shows a police vehicle 290 which has stopped a vehicle 291 witha driver suspected of impaired driving. The driver 292 has left the carand in FIG. 15 and an officer 293 is collecting information about thedriver with video and audio sensors in a device 294. The device doesappropriate preprocessing of the information and supplies it as input toa machine learning model trained for this use. The device informs theofficer with outputs signifying the probability that the driver is infact impaired and the officer decide whether to do additional testing.

The testing may consist of further data collection with the device ormay require

other types of testing. FIG. 16 shows two drivers which have been askedto stand on one foot as a test of their state. The device analyses theirmotions. Each driver is wobbling, but both sober and impaired driversmay wobble in this situation. The first driver 295 has his hands in hispockets which may contribute to the wobble.This driver is showing irritation with being stopped which maycontribute to a lack of effort to stand steadily and may affect theimpartiality of the officers judgement if evaluation is not made with anobjective system. The second driver 296 is attempting to appear soberand pass the test and trying hard to stand in a steady manner. When thesystem is trained on a wide variety of potential persons and situationsall of these factors will affect the trained behavior of the model.

An important reason to use an objective model is to prevent both thefact and appearance of unfairly judging persons with medical conditionsthat may appear falsely to some officers as evidence of impairment. Sucherrors can cause extreme embarrassment to law enforcement agencies. Thiscan be prevented by using a large number of examples of persons withsuch conditions in training the system. The training should be objectivein the sense of using training data with actual chemical or otherobjective test results. In this way, errors involving confusion ofbalance or speech impediments with effects of substance abuse can bereduced to very small levels.

Referring to FIG. 1T

A diagram of the process for developing the pattern matching softwarefor a typical embodiment of a machine learning system is shown. Thesoftware to be developed is a recurrent neural net with an input layer,a recurrent layer, an additional fully connected hidden layer and anoutput layer. Other more complex structures are also easily implementedbut this structure would be sufficient for the low level of accuracyneeded in many embodiments. This figure shows training in one level andthe steps would be repeated when training is accomplished in multiplelevels. An example of multiple level training is shown in FIG. 12 whichhas basic, general and specific levels of training prior to download tothe second platform for use and an “in-use” level after download. Thus,the system of FIG. 12 would use the steps of this figure up to fourtimes. Certain steps, such as checks for convergence, may be omitted forcertain levels.

The first step of the development at each level is to accumulate 310 adata set for training and testing. A device constructed similar todevice for use of the system can be used to collect data that matches inits use of sensors and preprocessing but the processor is programmed tocollect data from the sensors (such as item 280 of FIG. 13) and transmitit via the communication module to an outside data collection system. Adataset for the basic level of training (see 230 of FIG. 12) wouldtypically consist of data to teach components of sensor inputs such asidentifying edges in picture data and separating different soundsources. A dataset for the general level of training would train a modelto recognize particular events and objects such as persons, clothing orthe head of a lacrosse stick. A data set for the specific level wouldteach cases which distinguish between complying and non-complyingsituations. The data is usually divided into two sets with a largerportion for training and a smaller portion for accuracy testing. This isconsidered labeled data because it contains both input (sensor) data andthe desired output for that data.

In this and related embodiments, a step in the development which mightbe started in parallel with data collection is the design of anappropriate neural network. The sizing of the layers, the pattern ofinterconnection between layers and between gates within layers (such asthe addition of recurrency), and the setting of various factors in theneural net which are in addition to the factors and values (parameters)that are adjusted in training are collectively referred to ashyperparameters to distinguish them from the “parameters” which areadjusted in training the neural network. The hyperparameters areinitialized 311 to appropriate values. In some systems that are taughthyperparameters are adjusted during the course of training but aredistinct from trainable parameters because the

adjustments are on the basis of the progress of the training rather thanbeing direct functions of the data.

The next step is to initialize 312 the parameters which are to betrained. Appropriate initialization is necessary for reasonably rapidconvergence of the neural net. A number of techniques are widely knownto produce an initial set of values which generate good trainingprogress toward convergence.

The network is then trained 313 by passing data set items through thenetwork as implemented on a training processor. Because trainingrequires larger processing power and time than use of the network aftertraining special powerful processors are used for this step. Thetraining process adjusts the parameters incrementally on the basis ofthe output of the neural network. The hyperparameters specify themethods of calculating the adjustment to parameters. Generally, theoutput of the network is used to back propagate through the network toprovide further input to the adjustments. The items in the trainingportion of the dataset are used repeatedly while the convergence of thenetwork is observed 314 by processes in the training data processor.

If the convergence is judged 315 not to be adequate the training isstopped, the hyperparameters are adjusted 316, the neural network isreinitialized and the training process is repeated until satisfactoryconvergence is obtained. The smaller portion of the data set which hasbeen retained and not used for training is then passed 317 through theneural network (classified) and the output is checked 318 for accuracy.If accuracy is not sufficient for the goals of the particular systembeing developed then the net structure is made larger 319 and thetraining process is repeated until satisfactory accuracy is obtained.

The trained neural network is then downloaded 320 to the target deviceor made ready for next level of training.

(end material from PARENT AND GRANDPARENT applications)

Detailed Description for the CURRENT Application

Granularity

Coarse and Fine Granularity

Cameras, Microphones and other sensors produce signals which sample themeasured conditions at various resolutions over several differentdimensions. A camera can sample at a variety of pixel resolutions. Amicrophone or other sound sensor can take measurements at varioussampling rates. The reduction in sampling rates from the maximum rate inspace, time or some other dimension is called herein a reduction ingranularity. That is, this specification uses the definition ofgranularity with the convention common in the field of investing moregranularity indicates fine granularity and rejects the definition usedin photography where more granularity indicates coarse granularity.

Dimensions of Granularity

There are many different dimensions or axes which can have varied levelsof granularity. They include:

Pixel count of an image in either a single dimension or in multipledimensions. Even with a single sensor the sensitivity to low light canbe increased by combining sensor pixels to form a larger more sensitivevirtual pixel. This coarse granularity allows savings in energy forprocessing and for illumination.

Sampling rate for a sensor. In the case of audio sensor signals, thismeans that for coarser granularity samples are taken at greaterintervals. It is also possible to take bursts of samples with two ratesthe burst rate and the sample rate within a burst. For example, a burstof 100 samples at 20,000 samples per second could be taken 10 times asecond. The coarser granularity could be on either the burst rate or thesample rate.

Even if the samples are taken at the same granularity, the granularitycan be reduced in processing by using simpler approximate algorithm todo processing. This can be lossless or lossy.

Substitution of a sensor of finer granularity for a coarse sensor. Thefiner sensor may work in a different mode than the coarse sensor. Forexample acoustic sensors may be used to detect conditions for use ofactive illumination sensors working in radio or optical electromagneticmodes.

Data Size and Granularity

If the same area or time interval of observation is used and differentlevels of granularity, then the quantity of resulting data would belarger at the finer granularity. In some cases, the fine granularityobservation is made over a smaller area or time interval and the datasize remains the same. There may still be a saving of resources atcoarse granularity because of factors other than the number of samplestaken by a sensor.

Illumination and Granularity

Active illumination in any mode including light, radio or acoustic canresult in finer effective granularity. If the received returnedradiation or sound is increased is a possibility of increased depth insampling. The effective granularity becomes finer and insignificantdigits can become significant. Variation of illumination can consist ofeither using it only at finer granularity or by varying it's intensityat different modes. The type of sensors and even the general mode ofobservation can change. For example, acoustic sensors can be used forcontinuous monitoring and be augmented with light based imaging when amachine learning model indicates the desirability of such.

In the case of observation by visible, infrared or other light thegranularity can become finer in multiple ways. The use of an imagesensor multiple pixels with greater illumination can be can work atvarious resolutions determined by the

available light level. This is because the sensitivity can be increasedby combining the operation of multiple adjacent pixels into a singlelarger pixel. This is the principle of operation of low light modes onmany current consumer cameras.

Another way to increase the light sensitivity of a light image sensor isto take a longer sampling time resulting in a coarser granularity intime. If an image sensor works at a low light level the amplification ofthe sensor can be increased and a sample can be obtained, but the noisewill also be increased and the depth of the sample is reduced. That is,a sensor than can produce an 8 bit range of depth may only produce 4bits of depth in low illumination. Thus, there are at least threedimensions where reduced illumination can be used, pixel size orcombination, sampling frequency, and sampling depth.

Illumination in Other Modes

Illumination can be in other modes than with electromagnetic radiation.For example, the illumination may be with sonic pulses for systems thatuse active sonar. If illumination is with radio frequency energy (radar)then low level illumination can be used for coarse granularity and moreintense illumination can be used to refine the granularity.

Changing Mode to Improve Granularity

In many embodiments the same sensor system or sensor system that worksin the same mode is used for both coarse and fine granularityobservation. In some embodiments the sensor systems and mode ofoperation for different levels of granularity can be completelydifferent and work in different modes. For example, the coarsegranularity system may be a sonic system that detects movement and theapproximate area of the movement and the fine granularity system may bea high resolution camera with visible or infrared lighting.

Resources to be Conserved

Energy Consumption for Illumination.

One of the most important areas for conserving resources is to controlthe level of illumination provided. In a coarse granularity low level,natural or existing illumination may be sufficient to allow the requiredobservation. The illumination may be any form of energy includinginfrared, visible or ultraviolet light; sonic energy; radar or radiofrequency illumination; radiation; particle beams or some other form.The illumination itself or the energy to provide illumination may be theconservable resource.

Energy Consumption for Operation of Sensor

Many sensors themselves require energy for operation. This often varieswith the granularity of the sensor. There can be a substitution of afiner sensor with more resource use or the sensor itself may allowvarious modes with a tradeoff of granularity and energy use. Simplevideo sensors for cameras usually vary in the energy consumption withvariations in both the frame rate and in the pixel density. Someconsumer cameras are limited in the time they can operate at finegranularity by heating of the sensor and the related energy use. Thiscan also be reflected in battery life. High resolution video cameras aresometimes limited in the length of time that they can work at highresolutions and frame rates (fine granularity in time and pixel domains)by the heat that energy consumption produces. Cooling requirements are arelated resource that may need restriction on the operationalgranularity for conservation.

Energy Consumption or Other Resource Use in Transmission of Sensor Datafor Processing Elsewhere.

Many sensors especial video sensor produce large amounts of raw data. Ifthe data is to be processed by processors remote from the sensor, thisdata must be transmitted to the processor. This may be difficult in manysituation. One notable example is the data collected by probes in thefar reaches of space at the edge of the solar system. The transmissionitself can be the bottleneck in such systems.

Sensor, illuminator or other system component service life.

Limitations on hours of use may be the limiting factor in many systems.A familiar example is that ordinary incandescent light bulbs burn outafter a number of hours. Not turning illumination or other systems onuntil preliminary low granularity data indicates the need can reducethis problem.

Sensor Data Processing Processor Work.

Raw data from sensors can require substantial processing for use. Muchof that processing is scaled to the granularity. Each pixel or samplecan require individual processing. In other cases the amount ofprocessing can be much more than linear in its dependence on sampledensity the square, cube or higher power of the number of samples oreven in an exponential relationship.

Possibility of Detection of System by Adversarial Persons.

Operation of a system in fine granularity mode may require that thesystem expose itself to adversarial persons. For example, turning onradar or light illumination may announce presence. This may in turnrequire expenditure of resources to defend the system. If a vehicle isinvolved it may have to speed up or promptly leave the are consumingmore fuel. Additional jamming energy may also be required.

Avoidance of Disturbance of Persons, Animals or Systems by Illuminationor Use of the Sensor System.

Fine Granularity Data may require the acquisition or importation ofcomparison data with corresponding resource use.

Tradeoff Between Granularity and Resource Use.

Taking observations at finer granularity can result in production ofinformation that is of many uses. These include: identification of typeof an object observed, greater precision in location of objects,determination of speed or

movement of objects detection of additional objects, observation atgreater range, opportunity for further machine learning model analysisto determine details, and interactive information interchange with theobserved object.

Sensors

Sensor Types

Use of different types of sensors can greatly vary the requiredresources.

For example, acoustic or vibration sensors may require much less powerthan optical imaging sensors, especially if active illumination isneeded.

Description of Figures from the Current Application

FIG. 18

FIG. 18 depicts an embodiment which uses a video camera 410 to observein a remote area. The camera is mounted on a pole 411 and is powered bya small solar panel 412. There are lights 413, which could be continuousor have momentary flashing. The lights illuminate a target 414 to beobserved or analyzed by the camera. In the particular version of theembodiment which is depicted the target is a bird which may be part of aflock which could be a hazard to aircraft.

The camera is set up in the outskirts of an airport and is to allowaircraft to be warned to prevent bird strike damage. The space betweencamera's is large in this installation and use of the illuminationrequires substantial resources. The illumination is therefore only usedto do specific identification of the species of the birds as one thatare a threat. The unilluminated use at coarser granularity is used todetermine when to turn on the lights for a closer look. It is well knownthat combining pixels in a camera sensor can allow use in greatlyreduced illumination.

FIG. 19

FIG. 19 shows the field of the camera in the high resolution (finegranularity) mode 420 and in the low resolution (coarse granularity)mode 421 which allows use at low illumination levels. In the finegranularity case the details 422 picked up allow identification ofcritical details in determining the meaning of an observed object. Inthis case it allows determination if a bird is of a species thatproduced a bird strike threat to aircraft. The coarse granularityunilluminated image shows 4 activated pixels only 423 and is to beanalyzed by the machine learning model to determine if they signify apossible bird which would require expenditure of the energy to provideillumination for use of the fine granularity mode for furtherexamination.

FIG. 20

FIG. 20 shows a channel 430 in a waterway with a surface 431 and a sonarsystem 432 to monitor vessels passing through the channel. The sonarsystem listens passively to sounds 433 indicating passing objects suchas a submarine 434 or a whale 435. Application on an appropriate trainedmachine learning model to the received sounds can indicate with coarsegranularity the nature of the object making the sounds. The system maybe able to distinguish the approximate direction from which the soundsare coming. In some cases, here the submarine, there is a determinationthat a fine granularity analysis of the situation is warranted and sonicillumination 436 is turned on and the passive sonar is used in an activemode to determine the exact nature, location and other properties of theobject. The passive use conserves many potentially important things. Itconserves energy, prevents interference with other systems, reduces theinformation given to potentially adversarial persons and reducesinterference with marine wildlife. It

may be forbidden to use active sonar in the vicinity of marine mammalsand passive distinction between a vessel and a whale may be necessarybefore turning on sonic illumination.

FIG. 21

FIG. 21 shows a pair of directional multiple microphone systems 440 in awarehouse. It consists of an array of shotgun microphones such as 441each of which has a limited width of sensitivity. The two crossing beams442 produce a location with a coarse granularity proportional to widthof the beams in each dimension. In the depicted case the stackedmaterial in the warehouse 443 causes reflection of the sounds andsubstantial processing by means of a trained machine learning model isrequired to produce the best estimate of the source of a detected sound.Once the estimate is available a particular optical camera 444 isselected, turned on and aimed at the estimated position. A search mayensue and the source identified. Fine granularity in the optical systemmay be required for purposes such as identifying a person 445 as awarehouse employee rather than a intruder.

FIG. 22

FIG. 22 shows a system with three steps of granularity and training of acomputer learning model from data gained in the first step to aid theanalysis of the second step in triggering operation of the third step.An area of terrain 460 is shown with objects of three kinds 461, 462 and463. The object 463 is of interest for the detailed examination of thethird step. The first two steps are done by a unmanned aerial vehicle(UAV) 464 which overflys at an higher altitude 465 for the first stepand makes observations 466. The data gathered is analyzed with the useof a machine learning model trained in advance in the levels necessaryfor this purpose. This analysis enables either the same or another UAVto make low altitude flights 467 over as yet unclassified objects suchas 468 and with the aid of another machine learning model classifies theobjects with the aid of the finer granularity observations 469.

This step requires greater use of resources such as flight time and fuelbecause of the necessity to approach each object for close finergranularity observation rather than the bulk observation in the firststep.

In the third step a manned aircraft 470 examines 471 each objectselected in the second step with a sophisticated observation sensor withfiner granularity than in the earlier steps. Only the objects identifiedas being of the interesting kind such as 468 are examined by this veryresource intensive method.

FIG. 23

FIG. 23 is a diagram of the functional steps of the system of FIG. 22.The operation starts 480 and as a first step observes 481 the terrain ata high altitude with corresponding coarse granularity. A small number ofpasses is required because a wide area is observed from the distance andbecause of the limited number of passes a small use of resources isrequired. The data is analyzed 482 with a pretrained machine learningmodel. In the depicted case, the locations of objects on terrain isdetermined but from the distant course granularity observation theobjects cannot yet be classified by their type. A list of locations fromwhich the UAV to make close approach finer granularity observations isproduced by the analysis. On the basis of this list additionalobservations are made 484. These observations use additional fuel andvehicle time because they are made object by object instead of coveringa larger area containing multiple objects. A second stage of analysis485 with another pretrained machine learning model 486 yields a list oflocations for the final observations 487 at very fine granularity 488 tobe made with another system using at the expense of even greaterresources.

FIG. 24

FIG. 24 is a diagram of a more complex set of functional steps using theobjects and system of FIG. 22 with additional training of the machinemodel from data gathered by earlier observations. The operation starts490 and the UAV observes 491 a wide area of terrain at coursegranularity. The observations are analyzed 492 with a pretrained machinelearning model 493 producing both a list for the next observations anddata 494 for training a second machine learning model.

The 2nd model is trained 495 and the observations at finer granularity496

are performed. The trained model 497 is used to perform an analysis 498on the observations which results in production of a list of interestingobjects which is used to guide the very fine granularity and highresource use observations 499 of the piloted craft producing 500 veryfine granularity data concerning the interesting objects.

FIGS. 25 to 28 are described in the section on machine learning.

FIG. 29

FIG. 29 describes an embodiment where anti-surveillance devices ae usedto

detect an adversary device planted in an environment. The device 700,here called a ‘bug’ is concealed and transmits intercepted room soundsusing sophisticated techniques to avoid detection. It uses techniquessuch as operation below the noise, frequency hopping, burst transmissionand emulation of background noise to avoid detection. Several antennae701 in a room 702 gather signals in a broadband way and convey thesignals to a processor 703 which looks for signs that a bug may bepresent. This search is limited to the local signal processing power ofprocessor 702 and is low granularity with respect to any particularadversarial concealment scheme. Thus it may fail to detect the bug.

However, the processor is applying a machine learning model to look forsigns of a bug. After the pre-training accomplished prior to use onlarge scale training processors, the model has been trained on processor703 in the specific environment with various testing bug simulators. Inparticular, it has learned to detect signals from the correlation ofsignal delays between combinations of direct and reflected paths 704particular to the layout of the protected space. This enables detectionin some cases which are otherwise undetected. If the model on processor703 detects probability of a bug, it may be limited in the ability toanalyze the signals to confirm its detection, but can cause a largerprocessor system 705 located at a remote site which service manydetection devices to begin analysis of a larger sample of the signal toconfirm the detection and to derive further information

about the bug such as its exact location in the room and the types andcontent of the signals produced by the bug.

FIG. 30

FIG. 30 is a diagram of the operation and development and operation ofthe system of FIG. 29. A machine learning model is generated andpretrained 710 to identify potential signals from bugs. The model isdownloaded 711 to the local processor 703 (FIG. 29) which receivessignals from antennae 701 (FIG. 29). The local processor also gathersdata 712 from simulated bugs using the same antenna in the same layoutas in actual use. The model is then field trained 713 from the localdata.

The surveillance system composed of processor 703, antennae 701 and thetrained machine learning model is not placed in operation and scans forbugs 714 with its course granularity capabilities and resource use anddeciding 715 if a bug is suspected. If so then a larger processor 704 isnotified 716 and begins to take fine granularity, large volume data fromthe antennae. In a typical case, the local processor relays the data tothe larger processor which is off site. The local processor would thenhave sufficient data handling capabilities but not sufficient processingcapabilities for fine granularity analysis. The larger processor handlesthe fine granularity data and uses a larger more detailed machinelearning model with auxiliary signal processing using specializedhardware the analyze the signals to locate and classify the potentialbug. It accepts data from multiple antennae over a wide signal bus 720from the local processor.

Description of Machine Learning, Models and Training

A device to use gathered information to solve problems such as guiding aperson from a venue where a serious hazard exists requires verysophisticated

computer system to complex problems in interpreting incoming data andgenerating a plan for escape. In general, it is not feasible to discoverall of the rules and relationships necessary to solve that problem andto write a determinative computer program that produces a sufficientresult. However, methods have been developed and are wide and increasingto use a set of examples which is processed and used to product a set ofrules which working together can find answers suggested by the examples.A substantial set of examples and a large amount of processing arerequired but many people are trained and are being trained inapplication of well-known methods to implement this approach on a widevariety of problems.

There are limitations on the kind of problems that can be solved withthis approach, but the problem here is well suited for the approachbecause of the specific input data that is to be gathered and because ofthe specific kind of output that is required.

Machine learning as used here and in the Claims is a term for the typeof artificial intelligence that is implemented without explicitprogramming to generate solutions to the problems confronted. It isfocused on the development of solutions by having the effective programchange as it incorporates data in such a way that it converges on anability to produce the required solution.

Model

The term model as used in this specification includes representationsthat do not explicitly describe the system modeled but which aredesigned or trained to produce information based on the behavior of thesystem. It includes pattern matching system such as a neural networkwhich has been trained on data exemplifying the system. In that case themodel consists of a, probably huge, array of parameters that determinethe operation of the individual neurons in the neural net program.Training would work by systematically adjusting the values of theseparameters on the basis of the training data.

Machine Learning

Machine learning is a well-developed and understood method of creatingdevices that are capable of solving problems that defy development ofprogrammatic solutions that rely on understanding in detail the workingof the system to be analyzed. A famous example is the modern languagetranslation systems widely used on personal computing devices.Development of programs to translate languages has produced poor resultsbecause of the complex and subtle structure of human languages and thescale of the problem. But systems have been developed to be trained on alarge (possibly hundreds of millions) number of examples of languageusage. The trained models are then applied to an input in one languageand provide output which is very likely to be a satisfactory translationin another language of that input.

Machine learning systems are very different from computers runningprograms written to model problems to be solved. While theimplementation of a machine learning system may be made by means of acomputer program, this is not the only way to implement machine learningmodels. An array of analog devices (usually called gates) can implementthe model in a massively parallel way. Rather than containing a program,a machine learning system constructs a model which transforms an inputthrough a huge number of gates to produce an output which has astatistical meaning. The operation of the gates is modified in thetraining steps until the behavior of the model converges on a tendencyto produce desired results.

Machine Learning System Models

A machine learning system model or just “model” as used in thisspecification and in the claims is a large set of parameters representedas data or physically and arranged in such a way that they can beadjusted by a training

process based on a collection of data that represents the system beingmodeled. The model also allows inputs that represent a particular stateor set of states of the system to be analyzed by use of the model. Theuse of the model transforms the inputs into a set of outputs thatconstitute an analysis of the states being analyzed.

A model can be applied to a set inputs by means of an algorithm executedby a processor or by means of a physical analog device to perform thetransformation. The algorithm or device is only the means of evaluationand is distinct from the model which is the set of trained parametersand the structure in which they interact.

Training a Model

In this specification and in the claims the process of training a modelconsists of applying data representing possible inputs to the machinelearning system with the model in its current state of possibly partialtraining. The outputs of the system are used to generate incrementaladjustments to improve the transformation of the inputs into outputsbetter representing the desired behavior of the system.

The usual way to determine the adjustment to be made to the model foreach group of inputs presented is to calculate or measure the effect onthe outputs of each parameter in application of that set of inputs. Ifthe effect is favorable in providing outputs that correspond as wantedto the inputs then the parameter is very slightly augmented to improvethe overall behavior of the model as trained.

There are many ways to accumulate the data sets used for training. Oneway is to find or set up a large number of examples with known outcomesand collect the data from them. Another way is to write an algorithmwhich generates examples. The examples can be graded by people or thegeneration method may be able to predict the outcomes. Some problems areeasy to solve in reverse; i.e. a set of inputs may be easier to get froma assumed output than to find the output from a set of inputs. Forexample, to train a system to distinguish pictures of dogs from picturesof cats one can get pictures from public sources such as the internetand use humans to label the species depicted. That set can be used totrain a model which can test other pictures.

Convergence

The training process is continued for each item in the training setdata. Because it is important that training result in a stable andgradual progression of the model toward the desired behavior teach roundwhich uses the set of training data items only changes the model by asmall increment. The rounds are repeated many times and the results arecompared to data reserved for testing in order to measure theeffectiveness of training. If the structure of the model is well chosenthan parameters will converge on values that produce the desired outputsfor various input sets.

Training in Levels

Models can be arranged in levels both for training and for evaluation ofinputs. The application of the model to a set of inputs generatesoutputs that describe in a higher level of generality the meaning of theinputs. Those outputs can become inputs to further structure which is amodel for a more general transformation of the original inputs towardmeaningful outputs.

In this specification and in the claims, a level of training is thetraining of a portion of the parameters of a model to produce outputsthat are trained until a state of convergence is attained and madeavailable for input the next portion of the model. That is, distinctlevels are made distinct by separate training to convergence. It ispossible to simultaneously train multiple levels, but they are distinctlevels when they are separately tested for convergence. A level that isnot tested for convergence but which uses inputs from a level that hasbeen brought to convergence is a distinct level from the level providingthe inputs.

Typical models are in at least four levels. The first which here iscalled the Basic level takes raw sensor input and describes it in termsdirectly definable based on the input data. Examples would be detectionof edges from visual data and of tones, harmonics and burst timings foraudible data. The second level which is here called the General level isto identify objects and events from the output of the first level.Examples would be to detect a person crossing the path of the sensor oridentifying a sound as a gunshot or crowd noise. The third level, hereincalled the Specific level is to allow the model to identify actions andobjects appropriate to the purpose of use of the model. Examples of thislevel include model layers to implement steering or acceleration of avehicle or determination of compliance with a standard in a specifictype of situation. There is also a fourth level called the In-Use levelin many implementations. This level incorporates data collected while amodel is in use which modifies the model to allow evaluations at a latertime to take into account earlier inputs or evaluations where a seriesof evaluations is made.

Implementation of Training on a Processor with a Memory

Training requires a very large amount of processing to apply the largeamount of data in the training set repeatedly to incrementally cause themodel to converge on the desired behavior. If the adjustments from onepass through the data are too large, then the model may not converge ormay not allow the effects of all of the inputs to diffuse through themodel structure and correctly operate. For this reason, specialized verypowerful processors are used for training. They are not appropriate forincorporation in portable devices because of considerations of size andexpense.

Basic Training

In this specification and in the claims, basic training refers totraining which is used to interpret inputs from sensors or raw data fromdata sources to identify aspects of objects and actions treated asobjects that are implied by the data and too general in nature toidentify the potentially detected objects at this stage.

Examples include edge detection, categorization of sounds by location ofthe source, face detection, orientation in space, counting objects,elimination of backgrounds and many other general tasks ofinterpretation.

A portion of a machine learning model with this training can be used formany applications and could be supplied by a specialized developer. It'straining would be brought to convergence and the outputs supplied to thenext level of training when the model is used to evaluate inputs eitherfor further training of other levels or in actual use.

Data for General Training Describing the Area of Application of a Model

Data for the general level of training can be acquired by collecting anumber of real examples or by generating examples with a program toproduce examples and training data. In this and in other levels, it isoften much easier to produce a program for generating examples formachine learning than to determine specific rules to allow determinativenon-learning algorithms for evaluating rules designed for humanunderstanding.

Data for Training Concerning Compliance with a Standard

There is often available a set of examples to be made into training datafrom prior application of a rule set. For example, a dress code modelcould be made by using video collected over a period of time of peopleentering a venue. The videos could be graded by whether persons areadmitted or turned away by entry personnel. This would allow automaticgeneration of a training set by processing the videos.

Transferring a Trained Model

Levels of training of a machine learning model can be divided into twoclasses. The first class is those levels that require very large amountsof processing power and time to train. These typically use largetraining sets and are done before

other levels. They include in most cases the basic training levels whichare concerned into extracting interesting features from raw data usuallyfrom sensors and the general training levels which concern coordinationof features in sets of circumstances which are designed to encompass thespecific situation to be evaluated. These levels cannot be convenientlyhandled in real time and on a processor convenient to take into thefield to perform evaluations.

The second class of levels are those that must be performed after thespecific situation to be evaluated is determined. They must be performedin real time and on processors available at that time. The model trainedby the first class of levels can be transferred to a more convenientprocessor for the second class of levels of training.

Data for Testing in Particular Applications

After a model is trained in several levels and downloaded to a processorto use the model to evaluate situations, data must be collected with anappropriate sensor. The data is provided to the processer as input tothe model for an

evaluation to produce outputs. The outputs may have further non-machinelearning processing to produce a determination from the model in use.

Acquisition of Testing Data with a Sensor

A portion of the data collected or generated at each level is reservedfor testing. This data is not used for training to enable testing themodel without concern that the model might be effect only with thespecific cases used for training.

A Trained Model as a Special Purpose Machine

Once a model is trained and put into an environment that allows it toevaluate sets of input data the combination becomes a special purposemachine for making the determinations for which the model was trained.The feedback that is used to adjust parameters to produce desiredoutputs from inputs has created a

network that can operate on other inputs to produce similar results.This behavior has been tested and the machine can be put into use.

Figures Concerning Machine Learning

FIG. 25

FIG. 25 is a diagram of the structure of a simple recurrent neural net.A

neural net based on these principles is a preferred way to implement amachine learning model.

The layers of the model are sequenced as in the arrow 600. In thedepicted embodiment there are 4 layers. Sensors 601 detect data to beanalyzed an provide signals to a preprocessing unit 602 which appliesanalog and digital methods to simplify and quantify them for for inputfor evaluation by the machine learning system. Outputs 603 of thepreprocessing are supplied as inputs to the first layer 604 of themodel. The first layer is implemented in two sublayers 605 which arecompletely interconnected 606. Typical neural network models havemultiple sublayers in each layer and often have completeinterconnections. Each interconnection contains a parameter whichdetermines the strength of the interconnection. Each layer and sublayerconsists of a number of data structures called neurons Training adjuststhe parameters in small increments to cause the model to converge on thedesired behavior. A level of training works on a layer or group oflayers to produce convergence to the desired behavior for that level.

Connections between major layer structures 607 are often much moresparce and are designed to transferred information which is correlatedto patterns detected by the earlier layer. This layer is trained to do avery low level of pattern analysis producing patterns identifying groupsof related data and statistical representations of data. In a visiontype of system this might be a way to identify the edge of an object.

The second layer of the model 608 has a single sublayer. This layer hasrecursive connections 609 between outputs of neurons of the model whichallows the

model to represent time sequences. In practice this layer would haveother sublayers with much more complete connections between the neuronsof the layer. These sublayers are omitted to simplify the figure. Thislayer could be trained to work on the output of the first layer toidentify time structures of data.

The fourth depicted layer 611 is shown as being trained in an in-usetraining level. Data from a sensor is processed by a training program toallow more effective machine learning methods to be applied at that latestage by a training module 612 on the evaluating processor. Because ofthe limited time an processing power available for real time trainingthis is limited in scope but because of the extensive analysis alreadydone on the data by earlier layers of the model, a very simple layerwith simple training can make a major contribution to the results.

The outputs of the last layer are available then for non machinelearning processing, counting and use or display 613.

FIG. 26

FIG. 26 is a diagram of the process for developing the pattern matchingsoftware for a neural net such as in FIG. 25 is shown. The software tobe developed is a recurrent neural net with an input layer, a recurrentlayer, a additional fully connected hidden layer and an output layer.Other more complex structures are also easily implemented but thisstructure would be sufficient for the low level of accuracy needed inmany embodiments.

The first step of the development is to accumulate 620 a data set fortraining and testing. Several data sets are required as described forFIG. 25. One set of data is collected by a devices that sense details inquantity that might need to be used to develop the ability to analyzesparser data in final use. The processor is programmed to collect datafrom the sensors and transmit it via the communication module to anoutside data collection system. A substantial number of real orconstructed virtual objects are used for training and the resulting datais graded and categorized by manual or computer analysis and enteredinto the database.

The data is divided into two sets with a larger portion for training anda smaller portion for accuracy testing. This is considered labeled databecause it contains both input (sensor) data and the desired output forthat data (presence or absence of transfer to the next layer).

In this and related embodiments, a step in the development which mightbe started in parallel with data collection is the design of anappropriate neural network. The sizing of the layers and the setting ofvarious factors in the neural net which are in addition to the factorsand values (parameters) that are adjusted in training are collectivelyreferred to as hyperparameters to distinguish them from the “parameters”which are adjusted in training the neural network. The hyperparametersare initialized 621 to appropriate values. In some systems that aretaught hyperparameters are adjusted during the course of training butare distinct from trainable parameters because the adjustments are onthe basis of the progress of the training rather than being directfunctions of the data.

The next step is to initialize 622 the parameters which are to betrained. Appropriate initialization is necessary for reasonably rapidconvergence of the neural net. A number of techniques are taught toproduct an initial set of values which produced good training progress.

The network is then trained 623 by passing data set items through thenetwork as implemented on a training processor. Because trainingrequires larger processing power and time than use of the network aftertraining special powerful processors are used for this step. Thetraining process adjusts the parameters incrementally on the basis ofthe output of the neural network. The hyperparameters specify themethods of calculating the adjustment to parameters. Generally, theoutput of the network is used to back propagate through the network toprovide further input to the adjustments. The items in the trainingportion of the dataset are used repeatedly while the convergence of thenetwork is observed 624 by processes in the training data processor.

If the convergence is judged 625 not to be adequate the training isstopped, the hyperparameters are adjusted 206, the neural network isreinitialized and the training process is repeated until satisfactoryconvergence is obtained. The smaller portion of the data set which hasbeen retained and not used for training is then passed 627 through theneural network (classified) and the output is checked 628 for accuracy.If accuracy is not sufficient for the goals of the particular systembeing developed then the net structure is made larger 629 and thetraining process is repeated until satisfactory accuracy is obtained.

The trained neural network is then downloaded 630 to the target device,which is then ready for system testing 631.

FIG. 27

FIG. 27 shows the steps and structure of the basic information flow of adevice and its training. There is a timeline 640 which is divided intothree periods. First pre-download 641 which is the time prior todownload of the pre-trained machine learning model to a device for itsuse. The model is pre-trained during this time. The next time period ispre-use 642, the time for training from sensors associated with thedevice for use. which is used to train the escape device and storerelevant conclusions. The final time period is the use period 643.During this time collection of data and training continue but evaluationof the model and development of escape plans and instructions alsooccurs. The instructions are put to the output display transducers toallow the protected person to escape the venue.

During the pre-download period, the machine learning model isconstructed and initialized to parameter values suitable to alloweffective training 644. Pre-training 646 typically proceeds in twostages and uses two data sets. The first stage uses a generic data setand trains for general skills such as recognizing objects and otherentities such as edges from a camera sensor. The data set for this stagemay not be one specifically developed for this application but issuitable to the sensors and processing resources to be used in thedevice. The second stage used a data set

developed 645 specifically for this application which is based on realor virtual sensor data and use area analysis scenarios. Such data setswould find data concerning specific layouts and situations and dataconcerning typical area that can predict possible situations in actualareas to be especially relevant. The data sets are trained 646, usuallysequentially, into the model. Low level training. such as here used, iscomputation intensive; but it can be accomplished on a one-time basisusing substantial computing resources not needed by the finalsurveillance device. If the device is produced in mass quantities thepretraining results are loaded into each device in the process ofproduction. Additional data sets and training can be done prior to useof the device to adapt it to specific anticipated uses or locations.

During the pre-use period, the surveillance device or at least itssensors are in a use area. Sensors of the device collect 647 informationabout the area and objects or events in the area. Information is alsodownloaded 648 from various servers and beacons provided for this use orby outside sources. The sensor and server information is used to do morespecific training of the model. This can be accomplished by a processorin the device; but it may also be assisted by training resources locatedin servers away from the escape device. Sensor and server data can alsobe stored 650 to be used as an input in the later use stage. Thesurveillance device is then readied to begin use.

During the in-use period, the device is actively surveilling in a coursegranularity mode. The model may continue to be trained 651 from allinformation sources including stored data 652 and data that continues tobe collected from sensors 653 and servers 654. The model is evaluated655 based on its current training 252, stored data 652, continuingsensor data 653 and continuing server data 654. When the evaluation ofthe model 655 indicates the need the system transitions to the highresource use fine granularity mode 656

FIG. 28

FIG. 28 shows the training and operation steps of the model used torepresent data to be evaluated and the situation for evaluation.

This diagram is intended to show the relationships between differentlevels of training of a model each of which builds on the levels below.The flow of the training and application of the model are shown by thearrow 660. The main diagram of the figure shows the detailed steps. Thedescription is for a general neural net type of model. Other model typescan follow the same general flow and neural nets can incorporateimplementation details not shown. The model generally has layers whichare arranged in the same order as the training steps and when aparticular level of the model is being trained, data is evaluated by theearlier levels of the model which have already had training to provideinputs to the level being trained.

The model is initialized 661 with suitable values in a trainableparameter set. Abasic data set 662 with basic information is used toperform the first level of training 663 the model. The model wouldgenerally have multiple layers and the basic data set would be used totrain the earliest layers of the model. It would use data to allow theselayers to recognize or react to features such as edges in pictorial dataand sound impulses for audio data. This training would be applicable tomany applications of a machine learning system. It may be provided by asupplier of implementation and hardware systems and these layers may beacquired in an already trained condition by implementations ofapplications.

The second level of training 664 in the depicted embodiment is done witha second “General” data set 665. This data is selected to allow themodel to use inputs to recognize objects and entities relevant to theapplication of the model.

The general data set in the depicted embodiment is generated by acombination of data generation from a simulation 666 of generalapplications of the model and specific data gathered 667 for suchapplications. The applications at this level include recognition ofobjects such as persons, articles of clothing, signs and other itemsused to define and the movements made by sensors as they traverse ascene. Prior to the training at this level layers are typically added668 to the model to allow the training to take effect in facilitatinganalysis with the aid of the moved based on inputs processed bypreceding levels of trained model.

The third level of training 669 in the depicted embodiment is done witha second “Specific” data set 670. This data is selected to allow themodel to use inputs to recognize objects and entities relevant in acontext relevant to a particular application of the model. The generaldata set in the depicted embodiment is generated by a combination ofdata generation from a simulation of specific applications of the modeland specific data gathered for such applications. Typical informationused to generate a simulation at this level include many variations ofrelevant objects for the purpose of applying standards similar to theone to be implemented. Prior to the training at this level layers aretypically added 653 to the model to allow the training to take effect infacilitating analysis with the aid of the model based on inputsprocessed by preceding levels of trained model

After the model is trained through several levels, it is usuallydownloaded 674 from high powered training processors which are only usedto prepare the model to a smaller portable processor to execute themodel in actual use. To use the model to evaluate a situation data isgathered from the situation 675 by means of appropriate sensors andprepared to serve as an input 676 for the model. The model on the basisof (evaluating) the inputs generates 677 outputs 678 corresponding tothe action of the training on the parameters of the model.

In some more advanced implementations of the system, inputs and outputsare used to select 679 additional training for the model. Theinformation in the inputs and outputs can cause the download of sets ofparameters which can be added to the model or a limited training processsimilar to that used to develop the original model can be accomplishedby the evaluation processor.

I claim:
 1. A system to acquire data comprising: (a) a first subsystemadapted to make observations with a sampling sensor operating at acoarse level of granularity and a low sampling density which consumesresources at a limited level, (b) a processor to evaluate a machinelearning model trained to determine existence of a requirement for dataof finer granularity and a higher sampling density from the observationsof the first subsystem, (c) a second subsystem adapted to makeobservations with a sampling sensor at a level of granularity finer thanthe first subsystem and a higher sampling density than the firstsubsystem which is not limited to the level of resource use of the firstsubsystem wherein the second subsystem is used when the requirement forobservations of finer granularity and higher sampling density has beendetermined with the use of the machine learning model and theobservations of the sampling sensor of the first subsystem by theprocessor to exist and is not used when the requirement has not beendetermined by the processor to exist, (d) wherein the determination isbased on the detection by the first subsystem of an object of a specificgenus, and (e) wherein the observations of the second subsystem are usedto identify a species of the object in the genus.
 2. The system of claim1 wherein: the observations of the sampling sensor of the firstsubsystem include data from at least one of a gyro comprised in thefirst subsystem and an accelerometer comprised in the first subsystem.3. The system of claim 1 wherein: (a) the observations of the firstsubsystem comprise receipt of a signal and (b) the observations of thesecond subsystem are optical with a greater sampling rate than theobservations of the first subsystem.
 4. The system of claim 1 wherein:(a) the machine learning model has been trained on a second processordistinct from the processor to evaluate the machine learning model andsubsequently transferred to the first subsystem; and (b) the requirementfor observations of finer granularity is based at least in part on asecond determination that the sensor is in a moving vehicle and thesecond determination is based on data from at least one of a gyro and anaccelerometer comprised in the first subsystem.
 5. The system of claim 1wherein: (a) the machine learning model has been trained on a secondprocessor distinct from the processor to evaluate the machine learningmodel and subsequently transferred to the first subsystem; and (b) themachine learning model has been transferred to the first subsystemsubsequent to the training on the second processor; (c) the machinelearning model has been trained with data acquired by at least one of agyro and an accelerometer of the first subsystem subsequent to thetransfer; and (d) the requirement for observations of finer granularityis based at least in part on a second determination that the sensor isin a moving vehicle and the second determination is based on data fromat least one of a gyro and an accelerometer comprised in the firstsubsystem.
 6. The system of claim 1 wherein: the requirement forobservations of finer granularity and higher sampling density is basedat least in part on a determination from data from a sensor comprised inthe first subsystem that the sensor is in a moving vehicle.
 7. Thesystem of claim 1 wherein: the machine learning model has been trainedon a second processor distinct from the processor to evaluate themachine learning model and subsequently transferred to the firstsubsystem.
 8. A system to acquire data comprising: (a) a first subsystemadapted to make observations with a sampling sensor operating at acoarse level of granularity and a low sampling density which consumesresources at a limited level, (b) a processor to evaluate a machinelearning model trained to determine existence of a requirement for dataof finer granularity and a higher sampling density from the observationsof the first subsystem, and (c) a second subsystem adapted to makeobservations with a sampling sensor at a level of granularity finer thanthe first subsystem and a higher sampling density than the firstsubsystem which is not limited to the level of resource use of the firstsubsystem wherein the second subsystem is used when the requirement forobservations of finer granularity and higher sampling density has beendetermined with the use of the machine learning model and theobservations of the sampling sensor of the first subsystem by theprocessor to exist and is not used when the requirement has not beendetermined by the processor to exist, (d) wherein the determination isbased on a detection by the first subsystem of the presence of a person,and (e) wherein the observations of the second subsystem are used toidentify a specific person detected by the first subsystem.
 9. Thesystem of claim 8 wherein: the requirement for observations of finergranularity is based at least in part on a location determined by datafrom a sensor comprised in the first subsystem.
 10. The system of claim8 wherein: the requirement for observations of finer granularity andhigher sampling density is based at least in part on a determinationfrom data from a sensor comprised in the first subsystem the sensor isin a moving vehicle.
 11. The system of claim 8 wherein: the observationsof the first subsystem are acoustic and the observations of the secondsubsystem are optical with a greater sampling rate than the observationsof the first subsystem.
 12. The system of claim 8 wherein: (a) theobservations of the first subsystem are acoustic, (b) the observationsof the second subsystem are optical with a greater sampling rate thanthe observations of the first subsystem, and (c) the observations of thesecond subsystem are used to identify the specific person detected bythe first subsystem.
 13. The system of claim 8 wherein: (a) theobservations of the sampling sensor of the first subsystem include datafrom at least one of a gyro comprised in the first subsystem and anaccelerometer comprised in the first subsystem, (b) the observations ofthe second subsystem are optical with a greater sampling rate than theobservations of the first subsystem, and (c) the observations of thesecond subsystem are used to identify the specific person detected bythe first subsystem.
 14. The system of claim 8 wherein: the machinelearning model has been trained on a second processor distinct from theprocessor to evaluate the machine learning model and subsequentlytransferred to the first subsystem.
 15. The system of claim 8 wherein:(a) the machine learning model has been trained on a second processordistinct from the processor to evaluate the machine learning model andsubsequently transferred to the first subsystem; and (b) the machinelearning model has been transferred to the first subsystem subsequent tothe training on the second processor; and (c) the machine learning modelhas been trained with data acquired by the sensor of the first subsystemsubsequent to the transfer.
 16. A method of making observationscomprising: (a) making a first observation of an object using a samplingsensor at a coarse level of granularity and a low sampling densitychosen to conserve a resource; (b) using a processor with a machinelearning program with a machine learning model trained to use the firstobservation to determine the need to make a second observation at afiner level of granularity and at a higher sampling density than thefirst observation, (c) making the second observation of the object usinga sampling sensor at the finer level of granularity and at the highersampling density wherein the second observation requires a greater useof the resource than the first observation, (d) wherein an observationat a coarse level of granularity and a low sampling density detects thepresence of a person, and (c) wherein the observation at a finer levelof granularity and a higher sampling density is used to identify theperson.
 17. The method of claim 16 wherein: the need for an observationof finer granularity is based at least in part on the location of thesensor making the first observation.
 18. The method of claim 16 wherein:the first observation includes data from at least one of a gyro and anaccelerometer.
 19. The method of claim 16 wherein: the machine learningmodel has been trained on a second processor distinct from the processormaking the determination and subsequently transferred to the processormaking the determination.
 20. The method of claim 16 wherein: (a) themachine learning model has been trained on a second processor distinctfrom the processor making the determination and subsequently transferredto the processor making the determination; and (b) the determination fora need for observations of finer granularity is based at least in parton a determination that the sensor is in a moving vehicle using datafrom at least one of a gyro and an accelerometer.